Evaluating the Feasibility of RAG Models for Regulatory Compliance

Introduction

Collaborative Project Overview
- Led by Queen’s University Belfast researchers in the Finance and AI Research Lab - Industry Partner Funds Axis Ltd, a leading RegTech firm in Northern Ireland - Supported by UKRI through the UKFin+ programme.
- Aim: Evaluate the feasibility of Retrieval-Augmented Generation (RAG) models for enhancing regulatory compliance within FinTech.

Objectives

Project Objectives

1. Evaluate Retrieval-Augmented Generation (RAG) models for:
   • Regulatory reporting,
   • Risk assessments,
   • Monitoring processes.
2. Quantify economic efficiency through cost-benefit analysis and process mapping.
3. Address challenges in adaptability, interpretability, and usability.
4. Provide actionable insights for AI-driven compliance systems.

Towards a Unified Definition of AI

AI as Adaptive Capital

Artificial Intelligence (AI):
>A novel form of adaptive capital, capable of dynamic learning, autonomous decision-making, and task flexibility, designed to achieve specific objectives efficiently.

Key properties:
- Dynamic Efficiency: Self-improving through machine learning.
- Repurposability: Adaptable to diverse tasks.
- Scalability: Low marginal costs, enabling rapid expansion.
- Labour Dynamics: Both complement and substitute for human skills.
- Value Alignment: Ensures AI objectives align with societal and economic goals.

Economic Properties of AI

Key Implications of Adaptive Capital

1. Dynamic Efficiency: Continuous self-improvement enhances productivity.
2. Repurposability: Application to diverse regulatory compliance tasks.
3. Scalability: Processes scale at low additional costs.
4. Labour Impact: Augments high-skill tasks; automates repetitive processes.
5. Value Alignment: Critical for sustainable economic growth.

graph TD
    A[AI as Adaptive Capital]
    A --> B[Dynamic Efficiency]
    A --> C[Repurposability]
    A --> D[Scalability]
    A --> E[Labour Market Effects]
    B --> F[Productivity Growth]
    C --> F
    D --> G[Reduced Costs]
    E --> H[Redistribution of Skills Demand]

Implications for Policy and Economics

Aligning AI with Economic Goals

1. Value Alignment: Align AI’s objectives with human welfare.
2. Policy Challenges: Address labour displacement through upskilling and incentives.
3. Scalability & Risk: Leverage AI’s scalability while mitigating potential inequities.
4. Structural Changes: Long-term shifts in growth models and comparative advantages.

Methodology Overview

Hybrid Economic Analysis Framework

1. Process Mapping: Establish baseline workflows and resource allocation.
2. Cost-Benefit Analysis: Quantify direct and indirect cost savings.
3. Simulation Modelling: Test adaptability of RAG models to evolving regulations.
4. Stakeholder Feedback: Gather practical insights on usability and challenges.

Economic Methodology Overview

Hybrid Economic Analysis Framework

Preliminary Insights

1. RAG models are projected to reduce task time by 40%, saving ~20 staff hours weekly.
2. Early tests highlight improved interpretability for ESG reporting and risk assessments.
3. Stakeholders report an 8/10 usability score for compliance-related tasks.

Challenges and Limitations

Quantified Challenges:

• Data Quality: Parsing accuracy starts at 70%, requiring preprocessing. • Adaptability: Simulations show initial 3-week lag for updates; goal is 5 days.

Addressing Concerns:

• Use explainable AI (XAI) for output transparency. • Focus on modular integration with existing systems.

Ethical Considerations

• Fairness: Minimise bias via curated datasets. • Accountability: Establish clear documentation for decision trails. • Human Oversight: Support compliance professionals without replacing roles.

Future Work

1. Train domain-specific language models for finance.
2. Enhance retrieval accuracy using semantic search.
3. Explore long-term cost and efficiency gains via real-world trials.

Thank You!

Contact Information Dr Barry Quinn Queen’s University Belfast 📧 barry.quinn@qub.ac.uk